I guess this feature requires more thorough testing. Ideally, we should have a dedicated page on the documentation to describe precisely what the users would see and what they should do. And also to explain clearly the limitations of the current implementation.
To achieve this, we need a dataset of several GB so that the migration takes more than a few seconds and we can really interrupt it abruptly in the middle and then resume it.

I performed another test this morning with the following scenario:

1. Create a DynamoDB table with 1,000,000 random items
2. Start a migration to ScyllaDB Alternator with 10 scan segments, and a savepoints schedule of 20s
3. Observe the following lines in the logs:

   24/08/11 08:06:20 INFO DynamoDbSavepointsManager: Created a savepoint config at /app/savepoints/savepoint_1723363580.yaml due to schedule. Segments to skip: Set()
   24/08/11 08:06:39 INFO DynamoDbSavepointsManager: Marked segments [1] as migrated.
   24/08/11 08:06:39 INFO DynamoDbSavepointsManager: Marked segments [0] as migrated.
   24/08/11 08:06:40 INFO DynamoDbSavepointsManager: Created a savepoint config at /app/savepoints/savepoint_1723363600.yaml due to schedule. Segments to skip: Set(0, 1)
   

   These logs mean that the accumulator is correctly updated when segments have been completely migrated, and the content of the accumulator is correctly dumped into the savepoint files.
4. Kill the Spark job before the migration completed
5. Run the validator to check that the migration was indeed incomplete
6. Start another migration using the latest savepoint file as configuration
7. Observe that the segments that were marked as migrated were correctly skipped for the second migration
8. After the migration completed, run the validator to check that all the items are present

I then performed the following variation:

• instead of killing the migration job, I stopped the ScyllaDB service. In that case, the migration job continuously retries to write the data until the ScyllaDB service is up again. I could observe that the migration was not progressing anymore. Then I also killed the migration job, and restarted it from the latest savepoint file.

@tarzanek ping

All the PRs that this PR depends on have been merged. We now need to cut a new release of the emr-dynamodb-connector to unblock this PR. I will work on that ASAP.

Great, thanks!